Efficient representations of video sequences and their applications

Recently, there has been a growing interest in the use of mosaic images to represent the information contained in video sequences. This paper systematically investigates how to go beyond thinking of the mosaic simply as a visualization device, but rather as a basis for an eecient and complete representation of video sequences. We describe two diierent types of mosaics called the static and the dynamic mosaics that are suitable for diierent needs and scenarios. These two types of mosaics are uniied and generalized in a mosaic representation called the temporal pyramid. To handle sequences containing large variations in image resolution, we develop a multiresolution mosaic. We discuss a series of increasingly complex alignment transformation (ranging from 2D to 3D and layers) for making the mosaics. We describe techniques for the basic elements of the mosaic construction process, namely sequence alignment, sequence integration into a mosaic image, and residual analysis to represent information not captured by the mosaic image. We describe several powerful video applications of mosaic representations including video compression, video enhancement, enhanced visualization, and other applications in video indexing, search, and manipulation.